Enterprises are looking at ways of reducing costs and increasing efficiencies of their data processing system. A typical enterprise data processing system allocates individual resources for each of the enterprise's applications. Enough resources are acquired for each application to handle the estimated peak load of the application. Each application has different load characteristics; some applications are busy during the day; some others during the night; some reports are run once a week and some others once a month. As a result, there is a lot of resource capacity that is left unutilized. Grid computing enables the utilization or elimination of this unutilized capacity. In fact, grid computing is poised to drastically change the economics of computing.
A grid is a collection of computing elements that provide processing and some degree of shared storage; the resources of a grid are allocated dynamically to meet the computational needs and priorities of its clients. Grid computing can dramatically lower the cost of computing, extend the availability of computing resources, and deliver higher productivity and higher quality. The basic idea of grid computing is the notion of computing as a utility, analogous to the electric power grid or the telephone network. A client of the grid does not care where its data is or where the computation is performed. All a client wants is to have computation done and have the information delivered to the client when it wants.
At a high level, the central idea of grid computing is computing as a utility. A client of a grid should not have to care where its data resides, or what computer element processes a request. The client need only request information or computation and have it delivered—as much as needed whenever needed. This is analogous to the way electric utilities work; a customer does not know where the generator is, or how the electric grid is wired. The customer just asks for electricity and gets it. The goal is to make computing a utility—a ubiquitous commodity. Hence it has the name, the grid.
This view of grid computing as a utility is, of course, a client side view. From the server side, or behind the scenes, the grid is about resource allocation, information sharing, and high availability. Resource allocation ensures that all those that need or request resources are getting what they need. Resources are not standing idle while requests are left unserviced. Information sharing makes sure that the information clients and applications need is available where and when it is needed. High availability ensures that all the data and computation must always be there—just as a utility company must always provide electric power.
Grid Computing for Databases
One area of computer technology that can benefit from grid computing is database technology. A grid can support multiple databases and dynamically allocate and reallocate resources as needed to support the current demand for each database. As the demand for a database increases, more resources are allocated for that database, while other resources are deallocated from another database. For example, on an enterprise grid, a database is being serviced by one database server running on one server blade on the grid. The number of users requesting data from the database increases. In response to this increase in the demand for the database, a database server for another database is removed from one server blade and a database server for the database experiencing increased user requests is provisioned to the server blade.
Grid computing for databases can require allocation and management of resources at different levels. At a level corresponding to a single database, the performance and resource availability provided to the users of the database must be monitored and resources of the database allocated between the users to ensure performance and resource availability goals for each of the users are met. Between databases, the allocation of a grid's resources must be managed to ensure that performance and resource availability goals for users of all the databases are met. The work to manage allocation of resources at these different levels and the information needed to perform such management is very complex. Therefore, there is a need for a mechanism that simplifies and efficiently handles the management of resources in a grid computing system for database systems as well as other types of systems that allocate resources at different levels within a grid.
One such mechanism is the system described in Hierarchical Management of the Dynamic Allocation of Resources in a Multi-Node System Ser. No. 10/917,873, which uses a hierarchy of directors to manage resources at different levels. One type of director, a database director, manages resources allocated to a database among services that use a database and its database instances. The database director manages the allocation of database instances among the services. Another director, a cluster director, manages resources of a cluster of nodes between multi-node database servers hosted on the cluster. Yet another director, a farm director, manages resources allocated between clusters.
For each database, cluster, and farm director, one or more other standby directors sit idle waiting, ready to take over the responsibility of a database, cluster, or farm director that crashes. For example, multiple database instances for a database are running on multiple nodes. A database director runs on one of the database instances, while standby directors run on each of the other database instances. When a database director encounters a system failure, the standby director takes over as database director and assumes the crashed director's responsibilities. Directors that are currently performing the role and responsibility of a database director, cluster director, or farm director, rather serving as standbys, are referred to herein as active directors.
The directors are message driven, that is, they take action in response to messages transmitted to them from multiple sources. Specifically, each of the active directors receives messages notifying them of performance related events and conditions to which an active director may respond to by taking “remedial actions”, which are actions taken to adjust allocation of resources. While a director is a performing a remedial action, other such messages may be received. If the director crashes while performing the remedial action, and a standby director steps in, it is desirable that the standby director know or recover the messages the formerly active director had received and/or was acting upon and to respond to the messages as if the former director had not crashed.
Some of the remedial actions taken by a director are complex and have high latency, that is, require a relatively long time to complete once commenced. For example, in response to being notified that performance for a database has fallen below requirements, a cluster director may add an additional database instance for the database to a node already hosting another database instance. This operation may require that the already running database instance be brought down so that the additional database instance can be brought up. These operations can require a relatively long time to perform.
While a high latency action is being performed, other messages may be sent to the director. It is desirable that a director be able to respond to these notifications before the high latency action is completed.
Furthermore, a high latency action requires relatively numerous and complex operations, and is subject to greater risk of crashing. If a director performing a high latency action crashes, a long period of time may elapse before the failure is detected and a standby director steps in. It is desirable that a director undertakes a high latency action in a way that insulates the director from the risks of crashing and delay inherent to performing high latency operations.
Based on the foregoing, it is clearly desirable to have an approach for a group of entities taking action to adjust allocation of resources in response to messages, where the approach insulates the entities from the crashes caused by high latency actions and, in the case of a crash, allows a standby to recover messages sent to the crashed entity, including messages for which action had been commenced but not successfully completed.
Approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.